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ABSTRACT 



The purpose of this study was to determine whether a 
performance-based assessment system newly implemented in a graduate education 
(Masters of Education) program was related to the development of adult 
learners' cognitive skills. This assessment system required that learners 
develop and maintain a professional teaching portfolio showing evidence of 
the effective application of course knowledge in their P-12 classroom. 
Course-specific authentic assessments were included in this system, as well 
as a limited number of traditional assessment pieces. Participants completed 
a survey during the final class of each of 8 courses, for a total of 2,567 
survey responses grouped into 2 sets from June 1998 through May 1999. There 
were three primary findings. First, clusters or composites of instructional 
and curricular strategies tended to function together in relation to 
participants' perceived growth in cognitive skills. The same four composites 
were identified in each of the two survey sets. The second finding was that 
the curricular strategies clustered and, as a composite, strongly correlated 
with participants' perceived growth in cognitive skills. This strong 
correlation was also evident in both survey sets. The third finding was that, 
in both survey sets, assessment practices clustered and tended to function 
together in relation to participants' perceived skill growth. Although the 
study's results do not strongly support the use of performance-based 
assessment, both current educational theory and professional common sense do 
support its use. The survey is attached. (Contains 40 references.) ( SLD) 
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Within the educational community, the role of assessment is hotly debated. 
Whether the learner is a second grader in formal education or an adult in a professional 
development program, the questions are the same. How is learning effectively assessed? 
Can the process of that assessment enhance a learner’s education? 

Across the educational spectrum, from formal to non-traditional contexts and 
from GED programs to professional institutions, assessment strategies are being 
evaluated and redesigned. The goal: to develop assessment systems that both appraise 
student learning and contribute to the learning process. 

Study Purpose and Context 

The purpose of this study was to determine whether a performance-based 
assessment system newly implemented in a graduate education program was related to 
the development of adult learners’ cognitive skills. This assessment system required that 
learners develop and maintain a professional teaching portfolio showing evidence of the 
effective application of course knowledge in their P-12 classroom. Course-specific 
authentic assessments were included in this system, as well as a limited number of 
traditional assessment pieces. The research question of this study was - Does this newly 
implemented performance-based assessment system positively impact participants’ 
perceived growth in cognitive skills? 

The study context was the Graduate Studies in Education division of the College 
of Adult and Professional Studies, Indiana Wesleyan University. This College of Adult 
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and Professional Studies was established to meet the educational needs of the working 
professional adult. It offers its constituency graduate degree programs in business, 
nursing, and education. Evening, Saturday, and online courses are offered at more than 
60 off-campus locations and extensive off-campus support services are provided. 

In this study, the performance-based assessment system was implemented in the 
Masters of Education program. The professional teaching portfolio required of 
participants reflected their workplace performance, and was assessed by a peer, an on-site 
mentor, and a university advisor. Each program participant was an experienced educator, 
teaching in an elementary or secondary public school classroom. 

Theoretical Framework 

The theoretical framework of this study was the social learning theory of Johnson 
and Johnson (1994) and the assessment theory of Wiggins (1998). The Social 
Interdependence Theory of Johnson and Johnson assumed the way social 
interdependence was structured determined how individuals interact, which, in turn, 
determined behaviors. This theory provided a framework for the educational context of 
this study. In this study context, study groups and cohort communities supported 
learners’ acquisition of knowledge, attitudes and skills. This context was based upon the 
theoretical assumption that positive social interdependence and collective affective and 
cognitive interaction stimulate meaningful learning. 

Wiggins’ Assessment Theory (1998) stated that the assessment process enhances 
learning when designed as a coordinated set of authentic, performance-based learning 
activities. This theory provided the foundation for the redesign of the Masters of 



Education program’s assessment system. A theoretical assumption of this study was that 
learning would be positively impacted if assessment activities were aligned with 
performance-based course expectations and tied to workplace performance. 

Review of Literature 

A Nation At Risk (National Commission on Excellence in Education, 1983) was 
the first in a series on national policy documents that claimed that the performance of 
American schools was in decline. According to these documents, the decline was 
evidenced by dropping standardized test scores, unfavorable international comparisons, 
and inadequate workplace performance of public school graduates. Educational 
reformers recognized the need for significant changes in educational assessment practices 
(Archbald and Newman, 1992; Peterson, 2000; Wiggins and McTighe, 2000). Over the 
past two decades, calls for reform have addressed both large-scale institutional 
assessment initiatives and smaller scale professional and classroom assessment practices. 

An aspect of the movement to improve educational assessment practices has 
focused on authentic and performance-based assessments. Questions of consistency, 
validity and reliability have been raised (Charles and Mertler, 2002; CRESST, 1993). 
Assumptions about the contribution of such assessment activities to the learning process 
have been challenged (National Center for Postsecondary Improvement, 1999). 

However, educators concede that performance-based assessments represent “a new 
paradigm focusing on the performance of many effective strategies and practices required 
to become a successful and competent educational leader” (Meadows and Dyal, 1998, p. 



A key to the successful development of performance-based assessment 
instruments is stakeholder involvement. Green and Smyser (1996) observed that one- 
size-fits-all assessments reflect something done to students as opposed to something 
students do for themselves. They suggest that students play a more active role in 
assessment development. A shift to student involvement reflects the emerging attitude 
of educators that assessment should be an activity that makes a positive contribution to all 
stakeholders. Angelo (1994) stated, “If assessment is ever to improve substantively the 
quality of student learning, and not just provide greater accountability and efficiency, 
both faculty and students must become actively, continuously and personally involved” 
(P-4). 

The idea that assessment can improve the quality of student learning has fueled 
the move toward developing more authentic, performance-based assessments for all 
educational contexts — P-12 education, post-secondary education and adult education. 
These alternative assessments are designed to encourage students to think critically and 
to draw their own conclusions to complex problems (CRESST, 1993; Wiggins and 
McTighe, 2000). The function of performance-based assessments is to provide students 
with the opportunity to use prior knowledge, new learnings and appropriate skills to 
actively respond to complex tasks and solve significant and realistic problems. Learners, 
therefore, are viewed as active participants in the educational process, constructing 
meaning from new information and integrating their existing knowledge with new 
learning (Wittrock, 1991). 

The move toward more authentic assessments has encouraged educators to define 
the characteristics of high quality assessments and establish parameters for their 
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development. Angelo and Cross (1993) have noted that assessments should be student- 
center, teacher-directed and mutually beneficial. There is a growing consensus that new 
assessments should be multidimensional, measuring a broad range of abilities and 
interests (Courts and Mclnemey, 1993; Darling-Hammond and Falk, 1997; Peterson, 
2000). They should be linguistically appropriate and sensitive to multiple cultures and 
perspectives (Gordon, 2001; Linn, Baker and Dunbar, 1991). Performance-based 
assessments should also be value-driven, on-going, and integrated with the learning 
process (Astin, Banta and Cross, 1992; Peterson, 2000). 

The professional portfolio is a performance-based assessment instrument being 
implemented across professional contexts. Educators are finding that a professional 
teaching portfolio is a powerful tool for professional development (Van Wagenan and 
Hibbard, 1998) and for fostering problem solving and self-evaluation in pre-service 
teachers (Mokhtari and Yellin, 1996). Wolf (1996) has stated, “Although portfolios can 
be time consuming to construct and cumbersome to review, they can also capture the 
complexities of professional practice in ways that no other approach can. Not only are 
they an effective way to assess teaching quality, but they also provide teachers with 
opportunities for self-reflection and collegial interactions based on documented episodes 
of their own teaching” (p. 34). 

Performance-based assessment practices have been developed in response to the 
growing concern among educators that traditional assessment practices were not meeting 
the learning needs of students. By involving more stakeholders in the assessment process 
and focusing assessment on student learning, educators have made progress toward 
designing high quality authentic assessment instruments. Professional portfolios are 
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among the tools being effectively used to promote professional and student growth 
through self-reflection, problem-solving, and collegial decision-making. 

Method and Evidence 

Prior to implementing the redesigned assessment system, a program survey was 
conducted over a one-year period, June 1998 through May 1999. Participants in the 
Graduate Studies in Education program completed the survey during the final class of 
each of eight graduate education courses (N = 2,567). 

This survey was divided into two sections (Appendix A). Nineteen questions 
focused on instructional and curricular strategies. Students were asked to rate the 
effectiveness of these strategies on a five-point Likert scale from “not at all effective (1)” 
to “completely effective (5).” Ten questions focused on cognitive skill growth. Students 
were asked to rate the effectiveness of the completed course in helping them develop 
specific cognitive skills. Again, the ratings were on a five-point Likert scale ranging 
from “not at all effective (1)” to “completely effective (5).” The survey responses were 
then analyzed (Saunders, Batson & Saunders, 2000). 

The redesigned performance-based assessment system was implemented 
program-wide by the Fall of 2000. A program survey was then conducted over a six- 
month period, January through June, 2000. Participants in the Graduate Studies in 
Education program completed the same program survey of 29 questions as described 
above (N = 578). 

Using the statistical program SPSS, several statistical analyses were conducted 
using the data collected from both sets of surveys. These analyses tested for correlations 



between the independent variables (instructional and curricular strategies) and each of the 
ten dependent variables (cognitive skills) in each survey set. 

First, a Pearson correlation coefficient was run of each survey set. Next, a 
multiple regression analysis was conducted of each survey set. This analysis highlighted 
correlations between specific instructional and curricular strategies and specific cognitive 
skills. 

A factor analysis was then conducted on each survey set, using a varimax 
rotation. This analysis found four key clusters or composites of instructional and 
curricular strategies. 

A multiple regression analysis was conducted of each survey set, testing the 
correlation between the instructional and curricular composites (the independent 
variables) and each of the ten cognitive skills (the dependent variables). This multiple 
regression analysis was conducted again while controlling for course and for a selection 
of instructors. 

Interview evidence was also collected to provide a different perspective on the 
relationship between the program’s instructional and curricular strategies and 
participants’ perceived cognitive skill growth. Focus group interviews were held with 
five randomly selected cohort communities. 

Each survey set was analyzed separately. Analysis results of each set were then 
compared to one another to uncover relationships and patterns. Special attention was 
paid to the impact of traditional assessment strategies (1998 survey responses) and 
performance-based assessment strategies (2000 survey responses) upon perceived learner 
growth in cognitive skills. 
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Findings 



1. MULTIPLE REGRESSION ANALYSIS OF THE WHOLE SAMPLE 



A Correlation of the 19 instructional strategies with the 10 meta-skills 



METASKILL 


Instructor Strategy 


beta 


1. Worldview 


13. Faith evident 


.35 


2. Ethics 


13. Faith evident 


.21 


3. Lifelong learning 






4. Reading 


19. Textbook 


.30 


5. Critical thinking 






6. Problem solving 






7. Writing 


16. Homework 


.20 


8. Oral 






9. Finding information 


16. Homework 


.20 


10. Teamwork 


17. Discussion 


.20 



Beta coefficient with a significance of .01 or better. * indicates .05 or better. 
N: 2,567 responses 



2. FACTOR ANALYSIS 

A factor analysis with a varimax rotation was completed on the whole sample. Four key components or 
clusters of instructional strategies emerged. The “Composite Instructional Strategies” were the following: 
Composite Instructional Strategy #1: questions 1 - 7 = Andragogy 
Composite Instructional Strategy #2: questions 8-10, 18 = Assessment 
Composite Instructional Strategy #3: questions 15 - 17, 19 = Curriculum 



End of 
Course 
Survey 
Question 


Composite Instructional Strategy 


# 1 : 

Andragogy 


# 2 : 

Assessment 


# 3 : 

Curriculum 


# 4 : 

Faith 


1 


.68 








2 


.61 








3 


.70 








4 


.72 








5 


.72 








6 


.50 








7 


.57 








8 




.64 






9 




.84 






10 




.84 






11 










12 










13 








.69 


14 










15 






.69 




16 






.76 




17 






.68 




18 




.66 


, 




19 






.71 





N: 2,567 responses 



3 . MULTIPLE REGRESSION ANALYSIS OF THE FOUR COMPOSITE 
INSTRUCTIONAL STRATEGIES WITH THE TEN META-SKILLS 



M.Ed. Program Survey: June 1998 through May 1999 
N: 2,567 responses 



META-SKILL 


Andragogy 


Assessment 


Curriculum 


Faith 


1. Worldview 






21 


.34 


2. Ethics 


.21 




.33 


.21 


3. Lifelong 


.27 




.35 




4. Reading 






.50 




5. Critical thinking 


.24 




.40 




6. Problem solving 






.42 




7. Writing 






.37 




8. Oral 


.21 




.36 




9. Information 






.43 




10. Teamwork 






.30 




Beta coefficient with a significance of .01 or 1 


setter. * indicates .05 or better. 



M.Ed. Program Survey: January through June 2000 
N: 578 responses 



META-SKILL 


Andragogy 


Assessment 


Curriculum 


Faith 


1. Worldview 






.44 


.37 


2. Ethics 






.57 


.22 


3. Lifelong 


.16 




.49 


.17 


i 

t 






.63 


.12 


5. Critical thinking', 


.20 




.52 


.11 


6. Problem solving _ 


.17 




.51 


.13 


7. Writing 




.15 


.48 


.16 


8. Oral 


.21 




.47 


.15 


9. Information 






.59 


.16 


10. Teamwork 


.12 




.47 


.12 


Beta coefficient with a significance of .01 or 1 


setter. * indicates .05 or better. 
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Discussion 



The purpose of this study was to evaluate the effectiveness of a newly 
implemented performance-based assessment system in a professional education setting. 

The research question of this study was - Does this newly implemented performance- 
based assessment system positively impact participants’ perceived growth in cognitive 

skills? 

Three primary findings resulted from this study. First, clusters or composites of 
instructional and curricular strategies tended to function together in relation to 
participants’ perceived growth in cognitive skills. The same four composites were 

identified in each of the two survey sets. 

Although these composites were identified through factor analysis, each 

composite was predicted and supported by recognized educational theory The 
theoretical support for each composite hypothesized a positive relationship between 
specific strategies and student growth, explained how individual strategies complemented 
one another, and described the relationships between these instructional strategies. 

The second finding was that the curricular strategies clustered and, as a 
composite, strongly correlated with participants’ perceived growth in cognitive skills. 

This strong correlation between the curricular strategy composite and participants’ 

reported cognitive skill growth was evident in both survey sets. 

The context of this study was based on social learning theory and designed to be 

student-centered and collaborative. A large body of research confirmed that a 
collaborative context, coupled with an interactive, relevant curriculum, supports cognitive 
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skill attainment (Ahern-Rindell, 1998; Johnson, Johnson & Smith, 1994; Smith, 1994; 
Wilson, 1993). 

The third finding was that, in both survey sets, assessment practices clustered and 
tended to function together in relation to participants’ perceived cognitive skill growth. 

In the set of 1998-1999 survey responses, traditional assessment strategies correlated with 
no perceived growth in cognitive skills. That is, when analyzed through a multiple 
regression analysis, there was no significant correlation found between the program’s 
assessment practices of 1998-1999 and participants’ self-reported growth in cognitive 
skills. 

In the set of 2000 survey responses, performance-based assessment strategies 
correlated with perceived growth of only one cognitive skill. Although the correlation 
between the program’s assessment practices of 2000 and participants’ self-reported 
growth in cognitive skills was stronger than the 1998 survey responses, the only 
significant correlation was with the cognitive skill of communicating “effectively through 
writing.” 

Wiggins and McTighe (1998) have stated the effectiveness of assessment is 
ultimately determined by its achievement of desired learnings. Danielson claims it is 
through assessment that students advance their understandings (1996). In this study, 
traditional assessment practices were unrelated to participants’ reported cognitive skill 
growth. Therefore, according to current assessment theory, traditional assessment 
practices potentially inhibited participants from achieving desired learnings and 
advancing their understandings. 
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Also in this study, performance-based assessment practices were significantly 
related to participants’ perception of growth in only one of ten cognitive skills. The 
answer, then, to this study’s research question - Does this newly implemented 
performance-based assessment system positively impact participants’ perceived growth 
in cognitive skills? - must be, “Only slightly.” 

Several steps may be taken to increase the positive impact of this newly 
implemented performance-based assessment system. It is likely that staff development 
opportunities for professors implementing the new assessment system should be 
improved. McTighe (1996) stated that, if we expect students to learn from authentic 
assessment practices, then educators must engage in “performance-based instruction.” 
Creating authentic and meaningful learning experiences, encouraging learner decision- 
making, maintaining a sensitivity to multiple cultures and perspectives, and publicizing 
performance standards are instructional strategies suggested to enhance “performance- 
based instruction” (AAHE Assessment Forum, 1992; Dietel, Herman and Knuth, 1991; 
Gordon, 2001; Herrington and Herrington, 1998; McTighe, 1996). 

It is possible that participants need more time and training to develop the new 
skills required of performance-based assessment. Haller, Child and Walberg (1998) 
suggest that cognitive skills required for performance-based assessment be explicitly 
taught. Self-reflection (Burke, 1997), informed decision-making (McTighe, 1996), and a 
holistic view of the teaching and learning process (Cerbin, 1994) help equip the student 
to learn from the process of performance-based assessment. 

It also may be that professors and participants alike need to learn how to learn in 
this new educational context that is supported by performance-based assessments. When 
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performance-based assessments become foundational to the learning process. Postman 
and Weingartner’s “Immunization Theory of Education” is no longer operative. 
Revisiting and revising learned skills, knowledge and dispositions become the norm and 
learning becomes an ongoing activity. Shanker (1996) stated that performance-based 
assessment must offer “serious intellectual content, take explicit account of the various 
contexts of teaching, offer support for informed dissent, be ongoing, and imbedded with 
purpose ...” (p. 223). Learning to learn in this educational context may require time and 
creative effort. 

It is clear that further research is needed to answer the many questions raised by 
this study. Although this study’s results do not strongly support the use of performance- 
based assessment, both current educational theory and professional common sense do 
support its use. If assessment can enhance student learning, then it deserves continued 
dialogue and research. 



Conclusion 

The most current thinking among educators is that assessment reform is needed 
across educational contexts (Bond, 2000). Bond, Herman, and Arter (1994) have stated 
that we must change our assessment strategies to tie assessment design and content to 
new outcomes and purposes for assessment. A new purpose for assessment is to foster 
learning of worthwhile academic content, while providing educators with evidence for 
making decisions about curriculum and teaching (Porter, 2000). A new assessment 
design and content has been identified as authentic and performance-based, and is 
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regarded as more reflective of new curricular goals and methods of instruction (Geltner, 
1993; National Commission on Teaching and America's Future, 1996). 

The purpose of this study was to evaluate the effectiveness of a newly 
implemented performance-based assessment system in a professional education setting. 
This assessment system required that learners develop and maintain a professional 
portfolio showing evidence of the effective application of course knowledge in the 
workplace. 

Professional portfolios are being developed and evaluated across professional 
contexts. How can they effectively assess the strengths and areas for growth of the 
professional? How can they be used to encourage professional reflection and 
development? How can they enhance the learning process? Questions abound. Until 
research on the effectiveness of such assessment practices as portfolio development is 
conducted, presented, discussed, and evaluated, the questions will remain unanswered. 
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APPENDIX A 



End of Course Survey 




Core Group Course# Instructor 

Begi nnin g Date . Ending Date 



For proper processing please fiBinaO above information. 


n otat 
ttO 


very 

Utile 


some- 

what 


mostly 


com- 

pletely 


I Instructor 












1 . The instructor’s professional and academic experiences were 


i 


2 


3 


4 


5 


appropriate to teach and facilitate this course. 










2- The instructor made good use of the allotted class time 


1 


2 


3 


4 


5 


3. The instructor was effective in helping me share my professional 
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knowledge. 












4. The instructor treated me and my colleagues as professional adults. 
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5. The instructor was effective in encouraging nnrl motivating top nod 
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my colleagues. 












6. The instructor made effective use of the textbooks and the module. 
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7. The instructor made it clear how the subject matter could enhance my 
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professional growth. 












8. The instructor established clear criteria for grading my assignments. 
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(Assignments include in-class and out-of-class work that is graded) 












9. The instructor’s assessment of my assignments, to date, accurately 
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reflect my performance. 












10. The instructor provided timely, adequate feedback on the quality of 
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graded assignments. 












1 1 . The instructor set high standards for achievement in this course. 
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12. The instructor was accessible to me beyond the regular class period 
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13. The instructor’s Christian faith was evident 
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14. On the average, our class began and ended on time. 
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Curriculum 












1 . The module/ syllabus was written clearly. 
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2. lhe outside activities/ assignments helped me to understand and make 
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use of the knowledge I gained from this course. 












3. The in class activities/ assignments helped me to discuss key concepts 
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and experiences with other adult learners. 












4. The grading procedures measured what I actually learned, performed. 
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and produced 












5. I was able to understand the in formation in the textbook. 
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Ten Across Objectives 

Rate the effectiveness of this course in heipln^ me attain: 












1. 


The ability to see things from the perspective of a Christian world 
view. 
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2. 


Skill in ethical thought and action. 
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3. 
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4. 


The ability to read complex materials wife comprehension. 
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5. 


Skill in thin king critically concerning and performance (Le. 

reflective practice). 
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6. 


Skill in problem solving and decision making 
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7. 


The ability to communicate effectively through writing. 
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8. 


The ability to communicate effectively orally. 
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9. 


The ability to find needed information (sometimes called information 
literacy). 
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10. 


The ability to woric effectively in teams. 
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